{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How does Python import Modules?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we run a statement such as \n",
    "\n",
    "`import fractions`\n",
    "\n",
    "what is Python actually doing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing to note is that Python is doing the import at **run time**, i.e. while your code is actually running.\n",
    "\n",
    "This is different from traditional compiled languages such as C where modules are compiled and linked at compile time.\n",
    "\n",
    "In both cases though, the system needs to know **where** those code files exist.\n",
    "\n",
    "Python uses a relatively complex system of how to find and load modules. I'm not going to even attempt to describe this in detail, but we'll take a brief look at the main points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sys` module has a few properties that define where Python is going to look for modules (either built-in or standard library as well as our own or 3rd party):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where is Python installed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.prefix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where are the compiled C binaries located?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.exec_prefix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two properties are how virtual environments are basically able to work with different environments. Python is installed to a different set of directories, and these prefixes are manipulated to reflect the current Python location."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where does Python look for imports?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " 'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive\\\\python36.zip',\n",
       " 'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive\\\\DLLs',\n",
       " 'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive\\\\lib',\n",
       " 'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive',\n",
       " 'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive\\\\lib\\\\site-packages',\n",
       " 'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive\\\\lib\\\\site-packages\\\\setuptools-27.2.0-py3.6.egg',\n",
       " 'D:\\\\Users\\\\fbapt\\\\Anaconda3\\\\envs\\\\deepdive\\\\lib\\\\site-packages\\\\IPython\\\\extensions',\n",
       " 'C:\\\\Users\\\\fbapt\\\\.ipython']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically when we import a module, Python will search for the module in the paths contained in `sys.path`. \n",
    "\n",
    "If it does not find the module in one of those paths, the import will fail.\n",
    "\n",
    "So if you ever run into a problem where Python is not able to import a module or package, you should check this first to make sure the path to your module/package is in that list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At a high level, this is how Python imports a module from file:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* checks the `sys.modules` cache to see if the module has already been imported - if so it simply uses the reference in there, otherwise:\n",
    "* creates a new module object (`types.ModuleType`)\n",
    "* loads the source code from file\n",
    "* adds an entry to `sys.modules` with name as key and the newly created\n",
    "* compiles and executes the source code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing that's really to important to note is that when a module is imported, the module code is **executed**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's switch over to PyCharm (or your favorite IDE, which may well be VI/emacs and the command line!). All the files are included in the lecture resources or my github repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example shows that when we import a module, the module code is actually **executed**.\n",
    "\n",
    "Furthermore, that module now has its own namespace that can be seen in `__dict__`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we can see that when we `import` a module, Python first looks for it in `sys.modules`.\n",
    "\n",
    "To make the point, we put a key/value pair in `sys.modules` ourselves, and then import it.\n",
    "\n",
    "In fact we put a function in there instead of a module, and import that.\n",
    "\n",
    "Please **DO NOT** this, I'm just making the point that `import` will first look in the cache and immediately just return the object if the name is found, basically just as if we had written:\n",
    "\n",
    "`\n",
    "module = sys.modules['module']\n",
    "`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sys.modules['test'] = lambda: 'Testing module caching'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See, it got the \"module\" from sys..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function __main__.<lambda>>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Testing module caching'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 3a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we look at a simplified view of how Python imports a module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use two built-in functions, `compile` and `exec`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `compile` function compiles source (e.g. text) into a code object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `exec` function is used to execute a code object. Optionally we can specify what dictionary should be used to store global symbols.\n",
    "\n",
    "In our case we are going to want to use our module's `__dict__`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 3b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is essentially the same as example 3a, except we make our importer into a function and use it to show how we technically should look for a cached version of the module first."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
